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ABSTRACT 

This dissertation explores issues involved in higher 
education admissions processes. It analyzes admissions and subsequent 
performance at the RAND Graduate School. Academic literature informs 
the presentation of statistical relationships among admissions 
criteria, admissions committee ratings and performance measures. 
Transformation techniques provide the foundation for a discussion of 
the merits of alternative ways of thinking about selection, 
performance measurement, and prediction. While the RAND Graduate 
School admissions committee implicitly gives great importance to 
Graduate Record Examination (GRE) scores, different performance 
measures correlate most strongly with different selection criteria, 
complicating the establishment of screening rules. The empirical 
results show the statistical significance of several selection 
criteria in predicting a variety of measures of student performance 
and compare the significance of those criteria to the significance of 
quantified committee-member ratings in predicting the same 
performance measures. This report shows several techniques for 
transforming measures of selection criteria, ratings and performance, 
in order to discuss the appropriateness of relative and absolute 
measures. (Contains 113 references.) (MAH) 
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James R. Vernon 1 



This report fulfills the RAND Graduate School dissertation requirement for the doctoral program in public 
policy analysis for the author, a Graduate Fellow on leave of absence. 



The Role of Judgment in Admissions 
James R. Vernon 



This report explores issues involved in higher education admissions processes. It analyzes 
admissions and subsequent student performance at the RAND Graduate School. The 
academic literature on this problem informs the presentation of statistical relationships 
among admissions criteria, admissions committee ratings and performance measures. 
Transformation techniques provide the foundation for a discussion of the merits of 
alternative ways of thinking about selection, performance measurement and prediction. 

This research supports several conclusions: 

Admissions committees confront conflicting objectives. Committee decisions may show 
inconsistency across applicants, partly because selection combines individual judgments 
through persuasion and consensus-building. Furthermore, quantified measures of 
committee members' ratings may not fully reflect their true judgments about the 
desirability of applicants. 

Accepted applicants typically receive more ratings than rejected applicants. Moreover, 
committee members sometimes discuss applicants with each other and see other members' 
ratings before making their own judgments. Nevertheless, including the evaluation of an 
additional judge in average ratings increases the correlation between average ratings and 
both the admission decision and some performance measures. 

While the RGS admissions committee implicitly gives great importance to GRE scores, 
different performance measures correlate most strongly with different selection criteria. 
This complicates the establishment of screening rules. 

Transforming ratings and grades increases the correlation between them, and it increases 
the explained variance in admissions decisions. Thus, both admissions and later 
performance evaluation depend at least partially on a relative rating of students. Such 
relativism implies that admissions decisions, which depend at least in part on prediction of 
future performance, affect how that future performance will be measured. 

The empirical results show the statistical significance of several selection criteria in 
predicting a variety of measures of student performance. They also compare the 
significance of these criteria to the significance of quantified committee-member ratings in 
predicting the same performance measures. Furthermore, this report shows several 
techniques for transforming measures of selection criteria, ratings and performance, in 
order to discuss the appropriateness of relative and absolute measures. 

Although this dissertation focuses on data from RGS, it presents a framework for thinking 
about problems associated with admissions that general audiences may find useful or 
enlightening. 
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PREFACE 



This dissertation resulted from a debate among members of the admissions co mmi ttee at 
the RAND Graduate School. Committee members view the process as very time consuming and 
lack consensus about the appropriate criteria for admission. Some members emphasi ze Graduate 
Record Examination (GRE) scores when evaluating each applicant for admission. Others 
emphasize different attributes, such as previous grades. Committee members not only disagree 
over which factors to emphasize in selecting applicants, but they also disagree about what those 
factors mean. Some members also believe that the admissions process operates inefficiently. 

The research for this dissertation began as an attempt to find empirical relationships 
between a few selection criteria, such as GRE scores and previous grades, and aHmittpH 
applicants' subsequent grades at RGS, with the hope of finding useful screening criteria. This 
empirical analysis, however, raised as many questions as it answered. This research broadened 
to consider a multitude of empirical models. These reflect subtleties of the admissions process as 
well as complexities in performance measurement. In addition, some of these statistical models 
led to a consideration of philosophical issues in educational selection. The empirical results 
show the statistical significance of several selection criteria in predicting a variety of measures of 
student performance. They also compare the significance of these criteria to the significance of 
quantified committee-member ratings of applicants in predicting the same performance 
measures. Furthermore, this report evaluates several techniques for transforming measures of 
selection criteria, ratings and performance, in order to discuss the appropriateness of relative and 
absolute measures in the context of admissions committee decision making. 

Although the data for this dissertation come only from RGS admissions records and 
student transcripts, the implications of this research should interest a wide audience. Decisions 
such as hiring or promotion, for example, resemble the admissions decision insofar as scarce 
resources force the selection of fewer individuals than the number of applicants for a position. 
Predicting individual performance usually plays a significant role in such decision processes. 
While describing the admissions process at RGS, this dissertation presents a framework for 
thinking about the variety of problems associated with the process. 
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SUMMARY 



This report explores issues involved in the selection process in higher education. The 
problem requires an understanding of the objectives of selection, which must relate to the 
objectives of an institution. Questions of what to teach also influence who to select, and vice 
versa. Considering these questions fosters greater appreciation for the selection task. 

This paper considers that task by analyzing the admissions process and subsequent 
student performance at the RAND Graduate School (RGS). Predicting how well each applicant 
will perform represents just one of the difficult issues which admissions committees consider. 
To put this matter in perspective, this paper also discusses other issues in selection (such as 
predicting later life performance) as well as broader institutional objectives. The literature on 
selection and performance prediction informs the presentation of statistical relationships among 
various selection criteria, admissions committee ratings and measures of performance. 

This dissertation also examines the analytical methods used to estimate such statistical 
relationships. Much of the literature on selection and performance prediction reflects a 
straightforward approach to measurement of selection criteria, ratings and performance. This 
research highlights transformation techniques that may provide additional insights into the 
relationships among these measures. Rather than treat such measures as strict absolutes, this 
work attempts to show the impact of treating them in a relative context. It does so by 
considering various measures against relevant group averages and standard deviations. These 
transformation techniques provide the foundation for discussing the merits of alternative ways of 
thinking about selection, performance measurement and prediction. 

This analysis employs a broad array of statistical models covering admissions committee 
ratings, the a dmi ssions decision and a variety of subsequent student performance measures. The 
integration of these models provides a way to consider specific issues in the more general context 
of the academic environment. This context enables direct comparison of how specific factors 
(for example, GRE scores) influence committee ratings against how those same specific factors 
influence a particular performance measure (for example, attrition). Although previous research 




explores many of the specific issues covered in this report, this dissertation provides data from a 
single institutional environment to consider them all simultaneously. 

This research supports several conclusions. These conclusions apply not only to specific con- 
cerns at RGS but also to admissions and performance measurement policies at other institutions. 

Admissions committee ratings may not fully reflect decision makers' true judgments about 
the desirability of each applicant. The imperfect fit between overall committee ratings and 
actual committee admissions decisions at RGS demonstrates this point This cannot reflect mere 
uncertainty. If ratings represented decision makers’ best assessments of the desirability of each 
candidate, then actual decisions should follow those ratings (however unce rtain ) quite closely. 
However, actual admissions decisions show that the admissions committee makes judgments that 
do not strictly follow their ratings of individual candidates. Such judgments involve more than 
mere predictions of performance. This helps explain the imperfect correlations among selection 
criteria, ratings and performance measures. 

GRE Quantitative score has broad significance in predicting RGS outcomes, from ratings 
to admissions decisions to several performance measures. However, different performance 
measures correlate most strongly with different selection criteria and have varying 
predictability. Moreover, ratings have predictive value for RGS outcomes even after adjusting 
for criterion measures. Including the evaluation of an additional judge in average ratings 
increases the correlation between average ratings and both the admission decision and some 
performance measures, at least for the first few judges. This reinforces the role of judgment in 
admissions, independent of objectively measurable selection criteria. 

Transformation techniques demonstrate that treating ratings and performance measures 
as strict absolutes overlooks an important aspect of the issues involved in selection. 
Transforming ratings increases their ability to explain variance in admissions decisions. The 
improvement comes from grouping the ratings of each individual admissions committee 
member. Normalizing each rating against these group distributions transforms them to relative 
measures. (This normalization uses both a standard normal distribution as well as the 
distribution of GRE scores in each group as the distribution for the transformed measures.) In 
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addition, transforming grades increases the predictability of GPA. These transformations group 
the grades of each individual course. Normalizing each grade against these course distributions 
transforms them to relative measures. (This normalization also uses both a standard normal 
distribution as well as the distribution of GRE scores in each course as the distribution for the 
transformed measures.) Thus, both the admissions decision and later performance evaluation 
depend at least partially on a relative rating of students. Traditional analyses of ratings and 
performance measures as strict absolutes do not account for this complexity. This analysis seems 
to reflect a more realistic model of the processes involved. 

These considerations complicate the establishment of screening rules. Institutions can 
address these issues by clarifying objectives, rationalizing decision processes and enhancing 
evaluation policies. Clarifying objectives and l inkin g them to specific performance measures 
may enhance decision makers' understanding of their task and the limitations of their predictions. 
Even without strict screening rules, the RGS admissions committee can use the empirical results 
in this research to help lower its workload. Ratings of candidates and evaluation of students 
along separate dimensions can at least make uncertainty more explicit, and perhaps lead to more 
accurate (although narrower) predictions. 



I. INTRODUCTION 



Spending for higher education in the U.S. surpasses $175 billion annually. 1 The number 
of graduate students exceeds two million, and the number of undergraduates exceeds 12 million. 2 
These students attend over 3,600 separate institutions. 3 How to select these students has received 
much attention, as debates about affirmative action and truth in testing show. 4 This dissertation 
illustrates a methodology for evaluating some of the choices implicit in the admissions decision. 

Policy Issues in the Selection of Graduate Students 

When the number of applicants exceeds its capacity, an institution must decide which 
individuals will most likely best serve its goals. Although quite common, this problem remains 
complex. Institutions face a multitude of possible objectives. Some of these objectives may 
conflict with each other, and the implicit or explicit relative importance of each may vary. 

In addition to uncertainty over objectives, decision makers must consider a variety of 
selection criteria and determine their relative importance. Typically, such selection criteria 
correspond to institutional objectives. 5 In other words, once decision makers dete rmin e an 
objective, they look for performance measures that suit that objective. They then look for ways 
to predict that performance measure. Institutions consider many criteria for their usefulness in 
predicting some aspect(s) of future student behavior, such as academic performance or 
achievements later in life. In contrast, some schools also use criteria, such as race or gender, for 
their usefulness in predicting the characteristics of future groups of students and their impact on 
society, as distinct from the prediction of future individual behavior. 

Regardless of the emphasis on various objectives and criteria, admissions 
usually confront explicitly the prediction of academic performance. This prediction uses limited 
information and cannot elimin ate uncertainty. (Even for simple performance measures such as 

Statistical Abstract pf the U . S. , U.S. Department of Commerce, Washington, DC, 1994 d 151 

: m, P-152. F ' 

3 Ibid., p. 151. 

See, for example, Ralph Nader, The Reign of ETS: The Coporation That Makes Up Minds, Allen Naim 

and Associates, Washington, DC, 1980; and Robert Klitgaard, Choosing Elites, Basic Books New York. 

NY, 1985. 

This analysis does not consider principal-agent models or other impediments to rational decision making 
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attrition, prediction involves uncertainty.) Nevertheless, the importance of performance among 
institutional objectives makes predicting it (as accurately as possible) a desirable part of the 
selection process. This report considers the problem of predicting performance in the context of 
selection objectives, the literature on selection and prediction, and the admissions process at the 
RAND Graduate School (RGS), a doctoral program in public policy analysis. 

Selection criteria vary among institutions and among decision makers within institutions. 
Analysis of applicant characteristics, ratings and admissions at RGS illustrates a methodology for 
understanding more clearly the selection process and performance prediction. Placing selection 
criteria and processes in this context fosters greater appreciation of the difficulty in determining 
(empirically or rationally) selection objectives and their roots in broader institutional objectives. 

The usefulness of these criteria in predicting academic performance gives admissions 
committees a measure of the uncertainty they face and how much their selection criteria c an 
reduce that uncertainty. Decision makers can use empirical models of performance prediction to 
develop screening criteria or ranking mechanisms to improve the selection process by making 
better predictions or reducing the need for decision making resources. One of these resources, 
the human effort of individual admissions committee members causes some to wonder wha t they 
contribute to the selection process. Empirical models of selection and performance prediction 
shed some light on the role of judgment in admissions co mmi ttee decisions. 

The measurement of performance, selection criteria and subjective ratings influences the 
formulation of models for both selection and performance prediction. Decision makers need to 
choose performance measures with an understanding of how their choice affects the predictive 
value of various selection criteria and of their subjective ratin g s. Furthermore, combining 
individual judgments, for both admissions committee decisions and composite performance 
measures, raises the question of whether decision makers have consistent standards of jud gm ent. 
To address some of these measurement issues, this analysis employs a variety of transformations 
for both subjective ratings and performance measures. Norm alizin g both ratings and 
performance measures against group distributions transforms them to relative measures. This 
analysis considers relative measures within groups such as candidates rated by an individual 

admissions committee member as well as students evaluated by an individual instructor. These 
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transformations enable a comparison of absolute and relative ratings and performance measures. 
In other words, the statistical models used in this analysis can account somewhat for possibly 
divergent standards of measurement held by individual decision makers for both admissions and 
performance measurement. 

This comparison of absolute and relative measurement suggests a conceptual approach to 
the selection process that accounts for the relationships among institutional objectives, selection 
criteria, subjective ratings, admissions decisions, student performance and faculty evaluations. 
By exploring the broad context for these issues, this dissertation creates a framework for thinkin g 
about selection and prediction, illustrating methodologies for analyzing these problems. These 
methodologies can help decision makers better understand the nature of the problems they face 
and allow them to make more informed choices among objectives, selection criteria, screening 
and ranking mechanisms, performance measures and resource uses. 

Organization of This Report 

Section II considers institutional objectives, which form the background for a discussion 
of selection criteria. Section HI describes the RGS admissions process and relevant aspects of 
other selection processes. Section IV reviews the academic literature on selection and 
performance prediction to explore concepts and methodologies and to frame expectations for the 
empirical analysis in the present study. Section V describes the data used in that empirical 
analysis. Section VI shows the relationship between criterion measures and co mmi ttee ratings, 
after which Section VII presents the results of analyzing admissions decisions, using both 
criterion measures and ratings as predictors. Sections VIII and IX show the results of predicting 
various performance measures at RGS. Section X discusses the conclusions supported by this 
research and suggests a conceptual approach to thinkin g about selection and prediction. 
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